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I. Real Partv in Interest 

The real party in interest is Hewlett-Packard Development Company, L.P., a Texas 
Limited Partnership having its principal place of business in Houston, Texas. 

II. Related Appeals and Interferences 

Appellant is not aware of any related appeals or interferences that will directly affect 
or be directly affected by or have a bearing on the Board's decision in the pending appeal. 

III. Status of Claims 

Claims 1-6, 8-37, and 39-65, which are the subject of this appeal, are pending. 

Claims 7 and 38 are canceled. 

Claims 1-6, 8-37, and 39-65 stand rejected. 

Appellant appeals all rejections of the pending claims 1-6, 8-37, and 39-65, 
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IV. Status of Amendments 

The amendments filed November 5, 2003, have been entered and acted upon by the 
Examiner. 

No amendments were filed after the final Office action dated August 26, 2004. 

V. Summary of Claimed Subject Matter 

In the following Summary, the citations in parentheses are representative of support 
provided in the application. 

A, Independent claim 1 

The aspect of the invention defined in independent claim I is a method of interfacing 
with a machine. In accordance with this method, at each of multiple capture times, a 
respective image is contemporaneously acquired from each of multiple fields of view 
defining an interactive space to create a respective set of contemporaneous images (page 7, 
lines 17-22; FIG. 3, block 40). An input target is detected in the acquired images (page 1, 
lines 23-24; FIG. 3, block 42), Two-dimensional coordinates of the input target detected in 
the acquired images are computed (page 9, lines 23-25; FIG. 3, block 64). A spatiotemporal 
input data structure linking each of the capture times to the computed two-dimensional 
coordinates of the input target in each of the contemporaneously acquired images in a 
respective one of the sets of contemporaneous images is constructed (page 9, line 31 - page 
10, line 2; FIG. 3, block 66). The spatiotemporal input data structure is processed to identify 
an input instruction (page 12, lines 3-5; FIG. 3 block 92). The identified input instruction is 
executed on the machine (page 15, lines 5-6; FIG. 3, block 138). 

B. Dependent claim 13 

Claim 13 depends from claim 1 and recites that the spatiotemporal input data structure 
is constructed in the form of a linked list of data records (page 9, line 31 - page 10, line 2; 
FIG. 3, block 66; claim 13 as originally filed). 
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C Independent claim 30 

The aspect of the invention defined in independent claim 30 is a method of interfacing 
with a machine. In accordance with this method, sets of contemporaneous images of an 
interactive space are acquired from multiple respecti ve fields of view (page 7. lines 1 7-22; 
FIG. 3, block 40). An input target is detected in the acquired images (page 7, lines 23-24: 
FIG. 3, block 42). Coordinates of the input target detected in the acquired images are 
computed (page 9, lines 23-25; FIG. 3, block 64). A spatiotemporal input data structure 
linking input target coordinates computed from contemporaneous images to respective 
reference times is constructed (page 9, line 31 • page 10, line 2; FIG. 3, block 66). The 
spatiotemporal input data structure is processed to identify an input instruction (page 12, lines 
3-5; FIG. 3 block 92). The identified input instruction is executed on the machine (page 15, 
lines 5-6: FIG. 3, block 138). The method also includes interpolating between fields of view 
to generate a synthetic view of the interactive space (page 10, lines 20-21 ; page 1 8, line 32 - 
page 19Jine20). 

a Independent™ 31 

The aspect of the invention defined in independent claim 31 is a system for 
interfacing with a machine that comprises multiple imaging devices and a processing module. 
The multiple imaging devices are configured to contemporaneously acquire, at each of 
multiple capture times, a respective image from each of multiple fields of view defining an 
interactive space to create a respective set of contemporaneous images (page 7, lines 1 7-22: 
FIG. 3, block 40). The processing module is configured to detect an input target in the 
acquired images (page 7, lines 23-24; FIG. 3, block 42), compute two-dimensional 
coordinates of the input target detected in the acquired images (page 9, line 3 1 - page 10, line 
2: FIG. 3, block 66), construct a spatiotemporal input data structure linking each of the 
capture times to the computed two-dimensional coordinates of the input target in each of the 
contemporaneously acquired images in a respective one of the sets of contemporaneous 
images (page 9, line 31 - page 10, line 2; FIG. 3, block 66), process the spatiotemporal input 
data structure to identify an input instruction (page 12, lines 3-5; FIG. 3 block 92), and 
execute the identified input instruction on the machine (page 15, lines 5-6: FIG. 3, block 
138). 
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E Independent claim 32 

The aspect of the invention defined in independent claim 32 is a machine-readable 
medium storing machine-readable instructions for causing a machine to perform the 
following operations. At each of multiple capture times, a respective image is 
contemporaneously acquired from each of multiple fields of view defining an interactive 
space to create a respective set of contemporaneous images (page 7 ? lines 17-22; FIG. 3, 
block 40). An input target is detected in the acquired images (page 7, lines 23-24; FIG. 3, 
block 42). Two-dimensional coordinates of the input target detected in the acquired images 
are computed (page 9, lines 23-25; FIG. 3, block 64). A spatiotemporal input data structure 
linking each of the capture times to the computed two-dimensional coordinates of the input 
target in each of the contemporaneously acquired'images in a respective one of the sets of 
contemporaneous images is constructed (page 9. line 31 - page 10, line 2; FIG. 3, block 66). 
The spatiotemporal input data structure is processed to identify an input instruction (page 12, 
lines 3-5; FIG. 3 block 92). The identified input instruction is executed on the machine (page 
15, lines 5-6; FIG 3, block 138). 

F. Independent claim 33 

The aspect of the invention defined in independent claim 33 is a method of interfacing 
with a machine. In accordance with this method, an image is displayed at a display location 
disposed between a viewing space and an interactive space, wherein the displayed image is 
viewable from a perspective in the viewing space (page 24, lines 8-13; FIG. 14). Images of 
the interactive space are acquired from a field of view directed toward the interactive space 
along an optical axis intersecting a central area of the display location (page 24, line 28 - page 
25, line 2). An input target is detected in the acquired images (page 7, lines 23-24; FIG 3, 
block 42; page 25, lines 10-12). Coordinates of the input target detected in the acquired 
images are computed (page 9, lines 23-25: FIG. 3, block 64). An input instruction is 
identified based on the computed input coordinates (page 12, lines 3-5; FIG. 3 block 92). 
The identified input instruction is executed on the machine (page 15 ; lines 5-6; FIG. 3, block 
138). 
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G. Independent claim 63 

The aspect of the invention defined in independent claim 63 is a Claim 63 (previously 
presented): A system of interfacing with a machine. The system comprises a display, at least 
one imaging device, and a processing system. The display is configured to present an image 
at a display location disposed between a viewing space and an interactive space, wherein the 
displayed image is viewable from a perspective in the viewing space (page 24, lines 8-13; 
FIG. 14). The at least one imaging device is configured to acquire images of the interactive 
space from a field of view directed toward the interactive space along an optical axis 
intersecting a central area of the display location (page 24, line 28 - page 25, line 2). The 
processing system is configured to detect an input target in the acquired images (page 7, lines 
23-24; FIG. 3, block 42; page 25, lines 10-12), compute coordinates of the input target 
detected in the acquired images (page 9, lines 23-25; FIG. 3, block 64), identify an input 
instruction based on the computed input coordinates (page 12, lines 3-5; FIG. 3 block 92), 
and execute the identified input instruction on the machine (page 15, lines 5-6; FIG. 3, block 
138). 

K Independent claim 64 

The aspect of the invention defined in independent claim 64 is a Claim machine- 
readable medium storing machine- readable instructions for causing a machine to perform the 
following operations: display an image at a display location disposed between a viewing 
space and an interactive space, wherein the displayed image is viewable from a perspective in 
the viewing space (page 24, lines 8-13; FIG. 14); acquire images of the interactive space from 
a field of view directed toward the interactive space along an optical axis intersecting a 
central area of the display location (page 24, line 28 - page 25, line 2); detect an input target 
in the acquired images (page 7, lines 23-24; FIG. 3, block 42; page 25, lines 10-12); compute 
coordinates of the input target detected in the acquired images (page 9, lines 23-25; FIG. 3, 
block 64); identify an input instruction based on the computed input coordinates (page 12. 
lines 3-5: FIG. 3 block 92); and execute the identified input instruction on the machine (page 
15, lines 5-6: FIG. 3, block 138). 
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Dependent claim 65 



Claim 65 depends from claim I that recites "further comprising interpolating between 
ones of the images contemporaneously acquired from ones of the fields of view to generate a 
synthetic view of the interactive space" (page 10, lines 20-21; page 1 8, line 32 - page 19, line 



VI Grounds of Rejection to be Reviewed on Appeal 

A. Claims 1, 2, 4, 5, 8-12, 27-29, 31, and 32 stand rejected under 35 US.C. 
§ 102(b) over Mack (U.S. 6,198,485). 

B. Claim 13 stands rejected under 35 US.C § 103(a) over Mack (U.S. 
6,198,485). 

C. Claim 3 stands rejected under 35 U.S.C. § 103(a) over Mack (U.S. 6,198,485) 
in view of Kumar (US, 6,204,852). 

D. Claims 6, 14, and 15 stand rejected under 35 U.S.C. § 103(a) over Mack (U.S. 
6,198,485) in view of Pryor (U.S. 7,042,440). 

E. Claims 14 and 16-26 stand rejected under 35 U.S.C. § 103(a) over Mack (U.S. 
6,198.485) in view of Schmalstieg (U.S. 6,842,175). 

F. Claims 30 and 65 stand rejected under 35 U.S.C. § 103(a) over Mack (U.S. 
6,198,485) in view of Turkowski (U.S. 5,926,190). 

G. Claims 33-37, 39, and 60-64 stand rejected under 35 U.S.C. § 103(a) over 
Deleeuw (U.S. 6,753,879) in view of Ellenby (U.S. 5,682,332). 

H. Claims 41-44 and 46 stand rejected under 35 U.S.C. § 103(a) over Deleeuw 
(US. 6,753,879) in view of Ellenby (US. 5,682,332) and Mack (U.S. 6,1 98,485). 

I. Claims 45, 47, and 48 stand rejected under 35 U.S.C. § 103(a) over Deleeuw 
(US. 6,753,879) in view of Ellenby (U.S. 5,682,332), Mack (U.S. 6,198,485), and Pryor 
(US. 7,042,440). 

J. Claims 47 and 49-59 stand rejected under 35 U.S.C. § 103(a) over Deleeuw 
(U.S. 6,753,879) in view of Ellenby (U.S. 5,682,332). Mack (U.S. 6, 198,485), and 
Schmalstieg (U.S. 6,842,175). 



20). 
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VII. Argument 

A. Claim rejections under 35 U.S.C. § 102(b) 

Claims 1 , 2, 4, 5, 8-12, 27-29, 31, and 32 stand rejected under 35 U.S.C. §1 02(b) over 
Mack (U.S. 6J 98,485). 

1, Applicable standards for sustaining a rejection under 35 U.S.C. § 102(b) 

The relevant part of 35 U.S.C. § 102(b) recites that "A person shall be entitled to an 
invention, unless - ... the invention was patented or described in a printed publication in this 
or a foreign country or in public use or on sale in this country, more than one year prior to the 
date of the application for patent in the United States." Anticipation under 35 U.S.C. § 
102(b) requires that each and every element of the claimed invention be present, either 
expressly or inherently, in a single prior art reference. EMI Group N. Am.. Inc.. v. Cvpress 
Semiconductor Corp.. 268 F.3d 1342, 1350 (Fed. Cir. 2001). Anticipation must be proved by 
substantial evidence. In reCrishu 393 F.3d 1253, 73 USPQ2d 1364 (Fed. Cir. 2004). 

2. Independent claim 1 
a. Introduction 
Independent claim 1 recites: 



Claim 1 (previously presented): A method of 
interfacing with a machine, comprising: 

at each of multiple capture times, contemporaneously 
acquiring a respective image from each of multiple fields of 
view defining an interactive space to create a respective set of 
contemporaneous images; 

detecting an input target in the acquired images; 

computing two-dimensional coordinates of the input 
target detected in the acquired images; 

constructing a spatiotemporal input data structure 
linking each of the capture times to the computed two- 
dimensional coordinates of the input target in each of the 
contemporaneously acquired images in a respective one of the 
sets of contemporaneous images; 
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processing the spatiotemporal input data structure to 
identify an input instruction; and 

executing the identified input instruction on the 
machine. 

As explained in detail below, the rejection of independent claim 1 under 35 US C § 
102(b) over Mack should be withdrawn because Mack neither expressly nor inherently 
discloses each and every element of the invention defined by the claim, 

b. The Examiner's position and Appellant's rebuttal to the Examiners position 

In support of the rejection of independent claim I, the Examiner has acknowledged 
that Mack does not expressly disclose the ''constructing" element of claim 1 (see page 3, lines 
7-10, of the final Office action). In an effort to make-up for this failure of Mack's disclosure, 
the Examiner has argued that (see page 3, line 10 - page 4, line of the final Office action; 
emphasis added): 

... Mack discloses determining gesture inputs based on the 
movements of objects within the interactive space (col. 4, line 
45 -col. 5, line 12). Such a gesture would require the svstem to 
determine and save the coordinates determined from the images 
in a data structure while knowing the time in which the 
coordinates are determined . Without time and coordinate 
information the gestures could not be determined by the 
system. Further, if a data structure saving both the location 
information and coordinate information from the different two 
dimensional coordinates was not created, the computer would 
be unable to manipulate the data for calculation of three- 
dimensional coordinates and determination of gestures for 
operation of the computer system. Therefore, the generation of 
a data structure linking the capture times with the two- 
dimensional coordinates for later processing to determine input 
instructions and executing the instructions is inherent within the 
input system of Mack so that gestures can be determined to 
execute processes within the computer system. 

Mack, however, neither expressly nor inherently discloses "at each of multiple 
capture times, contemporaneously acquiring a respective image from each of multiple fields 
of view defining an interactive space to create a respective set of contemporaneous images" 
and ''constructing a spatiotemporal input data structure linking each of the capture times to 
the computed two-dimensional coordinates of the input target in each of the 
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contemporaneously acquired images in a respective one of the sets of contemporaneous 
images/' as now recited in claim 1 . 

The made-up rationale giv en by the Examiner in support of the rejection of claim 1 
does not meet the language of the claim. Indeed, claim 1 does not recite determining and 
saving "the coordinates determined from the images in a data structure while knowing the 
time in which the coordinates are determined/' nor does it recite generating "the generation 
of a data structure linking the capture times with the two-dimensional coordinates for later 
processing to determine input instructions and executing the instructions/' Instead, claim 1 
recites "constructing a spatiotemporal input data structure linking each of the capture times to 
the computed two-dimensional coordinates of the input target in each of the 
contemporaneously acquired images in a respective one of the sets of contemporaneous 
images" (emphasis added). The rationale given by the Examiner in support of the rejection of 
claim 1 does not show that Mack's system constructs "a spatiotemporal input data structure 
linking each of the capture times to the computed two-dimensional coordinates of the input 
target in each of the contemporaneously acquired images in a respective one of the sets of 
contemporaneous images." Therefore, the Examiner has not established & prima facie case 
the Mack inherently discloses all the elements of claim 1. 

Moreover, in the cited section of Mack s disclosure, Mack discloses that the system 
300 tracks the three-dimensional movement of the users eyes and the user's head using 
three-dimensional depth, rotation, and translation parameters. The tracked 3-D data, 
however, does not link each of multiple capture times to the computed two-dimensional 
coordinates of an input target in each of the contemporaneously acquired images in a 
respective contemporaneous image set created by contemporaneously acquiring a respective 
image from each of multiple fields of view defining an interactive space, as recited in claim 
1 . Instead, the tracked 3-D data merely stores three-dimensional depth, rotation, and 
translation parameter values with the associated capture times. 

For example, Mack expressly discloses that in the process of computing the 3-D data, 
the system simply determines the object points PI, P2 in each stereo image pair and finds 
"the intersection of the lines connecting the camera sources and the corresponding image 
object points on the image planes" (col. 6, lines 31-34; also see col. 5, lines 13-64). This 
process is performed separately for each stereo image pair and involves, for each stereo 
image pair, inputting the constituent stereo images and outing the calculating 3-D coordinate 
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of the object. Contrary to the Examiner's position, there is no need whatsoever for Mack's 
system to construct ^a spatiotemporai input data structure linking each of the capture times to 
the computed two-dimensional coordinates of the input target in each of the 
contemporaneously acquired images in a respective one of the sets of contemporaneous 
images," as recited in claim I. 

For at least these reasons, the rejection of claim 1 under 35 U.S.C, § 102(b) over 
Mack should be withdrawn. 

3. Dependent claims 2. 4. 5. 8-12. and 27-29 

Each of claims 2, 4, 5, 8-12, and 27-29 incorporates the elements of independent 
claim I and therefore is patentable over Mack for at least the same reasons explained above 
in connection with independent claim L 

4. Independent claim 3 1 

Independent claim 31 recites features that essentially track the pertinent features of 
independent claim 1 discussed above. Therefore, independent claim 3 1 is patentable over 
Mack for at least the same reasons explained above in connection with independent claim 1. 

5. Independent claim 32 

Independent claim 31 recites features that essentially track the pertinent features of 
independent claim I discussed above. Therefore, independent claim 31 is patentable over 
Mack for at least the same reasons explained above in connection with independent claim L 

B, Claim rejections under 35 U.S.C. § 103(a) 

1 . Applicable standards for sustaining a rejection under 35 U.S.C. § 103(a) 

"A patent may not be obtained ... if the differences between the subject matter sought 
to be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to 
which said subject matter pertains." 35 U.S.C. §103(a). 

In an appeal involving a rejection under 35 U.S.C. § 103, an examiner bears the initial 
burden of establishing prima facie obviousness. See In re Riickaert 9 F.3d 1 53 1 . 1 532, 28 
USPQ2d 1955, 1956 (Fed. Cir. 1993). To support a prima facie conclusion of obviousness, 
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the prior art must disclose or suggest all the limitations of the claimed invention. 1 See In re 
Lovvrv. 32 F.3d 1579, 1582, 32 USPQ2d 1 031, 1034 (Fed. Cir. 1994). If the examiner has 
established a prima facie case of obviousness, the burden of going forward then shifts to the 
applicant to overcome the prima facie case with argument and/or evidence. Obviousness, is 
then determined on the basis of the evidence as a whole and the relative persuasiveness of the 
arguments. This inquiry requires (a) determining the scope and contents of the prior art; (b) 
ascertaining the differences between the prior art and the claims in issue; (c) resolving the 
level of ordinary skill in the pertinent art; and (d) evaluating evidence of secondary 
consideration. See KSR Infl Co. v. Teleflex Inc. . No. 127 S. Ct. 1727, 1728 (2007) (citing 
Graham v. John Deere. 383 U.S. 1, 17-18, 148 USPQ 459, 467 (1966)). If all claim 
limitations are found in a number of prior art references, the fact finder must determine 
whether there was an apparent reason to combine the known elements in the fashion claimed. 
See KSR . 1741. This analysis should be made explicit. KSR at 1 74 1 (citing In re Kahn . 44 1 
F. 3d 977, 988 (Fed. Cir. 2006): ^Rjejections on obviousness grounds cannot be sustained 
by mere conclusory statements; instead, there must be some articulated reasoning with some 
rational underpinning to support the legal conclusion of obviousness"). 

2. Claim 13 

Claim 13 stands rejected under 35 U.S.C. § 103(a) over Mack (U.S. 6,198,485). 

Claim 13 incorporates the elements of independent claim I and therefore is patentable 
over Mack for at least the same reasons explained above in connection with independent 
claim I. 

In support of the rejection of claim 13 under 35 U.S.C. § 103(a ) over Mack, the 
Examiner has taken the following position: (see § 4 on pages 5-6 of the final Office action): 



1 The U.S. Patent and Trademark Office has set forth the following definition of the 
requirements for establishing a prima facie case of unpatentability (37 CFR § 1 .56(b)(ii)): 



A prima facie case of unpatentability is established when the 
information compels a conclusion that a claim is unpatentable 
under the preponderance of evidence, burden-of-proof standard, 
giving each term in the claim its broadest reasonable 
construction consistent with the specification, and before any 
consideration is given to evidence which may be submitted in 
an attempt to establish a contrary conclusion of patentability. 
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The Examiner notes that the use of a linked list data structure to 
store input data would be a design choice for one skilled in the 
art. One skilled in the art would be motivated to utilize a linked 
list data structure because of the ability of a linked list to store a 
sequence of information in order with each piece of data being 
linked directly to the following (and sometimes preceding) 
data It would be an obvious choice for one skilled in the art to 
use a linked list to store a sequence of input positions 
associated in a sequence of time. 



The storage of Mack's 3-D input data in the hypothetical linked list data structure 
invented by the Examiner, however, does not meet the claim language, which requires "a 
spatiotemporal input data structure linking each of the capture times to the computed two- 
dimensional coordinates of the input target in each of the contemporaneously acquired 
images in a respective one of the sets of contemporaneous images" (emphasis added). 

For at least this additional reason, the rejection of claim 13 under 35 U.S.C § 103(a ) 
over Mack should be withdrawn. 

3. Claim 3 

Claim 3 stands rejected under 35 U.S.C. § 103(a) over Mack (U.S. 6, 198,485) in view 
of Kumar (U.S. 6,204,852). 

Claim 3incorporates the elements of independent claim 1. Kumar does not make-up 
for the failure of Mack to disclose or suggest the elements of independent claim 1 discussed 
above. Therefore, claim 3 is patentable over Mack in view of Kumar for at least the same 
reasons explained above in connection with independent claim 1 . 

4. Claims 6. 14. and 15 
a. Introduction 

Claims 6, 14, and 15 stand rejected under 35 U.S.C. § 103(a) over Mack (U.S. 
6,198,485) in view of Pryor (U.S. 7,042,440). 

Each of claims 6, 14, and 15 incorporates the elements of independent claim 1. Pryor 
does not make-up for the failure of Mack to disclose or suggest the elements of independent 
claim I discussed above (see page 1 4 of the Amendment dated June 2 1 1 2007). Therefore, 
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claims 6. 14, and 15 are patentable over Mack in view of Pryor for at least the same reasons 
explained above in connection with independent claim 1. 

5. Claims 14 and 16-26 

Claims 14 and 16-26 stand rejected under 35 U.S.C. § 103(a) over Mack (U.S. 
6,198,485) in view of Schmalstieg (U.S. 6,842,175). 

Each of claims 14 and 16-26 incorporates the elements of independent claim 1. 
Schmalstieg does not make-up for the failure of Mack to disclose or suggest the elements of 
independent claim 1 discussed above. Therefore, claims 14 and 16-26 are patentable over 
Mack in view of Schmalstieg for at least the same reasons explained above in connection 
with independent claim I . 

6. Claims 30 and 65 

Claims 30 and 65 stand rejected under 35 U.S.C. § 103(a) over Mack (U.S. 
6,198,485) in view of Turkowski (U.S. 5,926,190). 

a. Introduction 

Independent claim 30 recites: 



Claim 30 (previously presented): A method of 
interfacing with a machine, comprising: 

acquiring sets of contemporaneous images of an 
interactive space from multiple respective fields of view; 

detecting an input target in the acquired images; 

computing coordinates of the input target detected in the 
acquired images; 

constructing a spatiotemporal input data structure 
linking input target coordinates computed from 
contemporaneous images to respective reference times; 

processing the spatiotemporal input data structure to 
identify an input instruction; and 

executing the identified input instruction on the 
machine; and 

interpolating between fields of view to generate a 
synthetic view of the interactive space. 
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The Examiner has given the following explanation in support of the rejection of claim 
30 (see § 8 on page 10 of the final Office action; emphasis added): 

Regarding claim 30, as discussed above, Mack discloses all 
steps of inputting coordinate information, determining gestures 
based on the information and executing instructions based on 
the input gesture information. Further, Mack discloses 
changing the viewpoints of displayed objects based on the input 
gestures (col. 5, lines 1-12). 

However, Mack does not expressly disclose generating 
different viewpoints based on interpolating between different 
fields of view. 

Turkowski discloses a method of generating different 
viewpoints of an object for display based on interpolation 
between two known fields of view (abstract; Fig. 3; col. 4, lines 
12-53). 

At the time of invention it would have been obvious to modify 
the teachings of Mack with the teachings of Turkowski. The 
viewpoint modification methods of Mack could be changed to 
include generation of new viewpoints based on the 
interpolation methods of Turkowski. The rationale would have 
been to include known methods for generating new synthetic 
viewpoints from predefined viewpoints when viewing images 
on a computer system and moving between new viewpoints 
when viewing the images. Thus, it would have been obvious to 
modify the teachings of Mack with the teachings of Turkowski 
to produce a method of interfacing with a machine as described 
in claim 30. 

In this explanation, however, the Examiner has not shown that Mack in view of 
Turkowski discloses "computing coordinates of the input target detected in the acquired 
images" and "constructing a spatiotemporal input data structure linking input target 
coordinates computed from contemporaneous images to respective reference times/ 5 as 
recited in claim 1 . Therefore, the Examiner has not established a prima facie case that claim 
30 is obvious over Mack in view of Turkowski. For at least this reason, the rejection of claim 
30 over Mack in view of Turkow ski should be withdrawn. 

Moreover, neither Mack nor Turkowski discloses or suggests "computing coordinates 
of the input target detected in the acquired images" and "constructing a spatiotemporal input 
data structure linking input target coordinates computed from contemporaneous images to 
respective reference times/' as recited in claim 1. 
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Mack expressly discloses that in the process of computing the 3-D data, the system 
simply determines the object points PI, P2 in each stereo image pair and finds "the 
intersection of the lines connecting the camera sources and the corresponding image object 
points on the image planes" (col. 6, lines 31-34; also see col. 5, lines 13-64). This process is 
performed separately for each stereo image pair and involves, for each stereo image pair, 
inputting the constituent stereo images and outing the calculating 3-D coordinate of the 
object. This process does not expressly nor inherently involve computing coordinates of the 
input target detected in the acquired images and constructing a spatiotemporal input data 
structure linking the input target coordinates computed from contemporaneous images to 
respective reference times. Indeed, the object points P 1 , P2 in each stereo image pair are not 
part of a spatiotemporal input data structure linking the object points PI, P2 computed from 
contemporaneous images to respective reference times. Instead, the output 3-D data is 
associated with respective capture times. 

For at least these reasons, the rejection of claim 30 under 35 U.S.C. § 103(a) over 
Mack in view of Turkowski should be withdrawn. 

The rejection of claim 30 over Mack in view of Turkowski also should be withdrawn 
for the following additional reason. 

The Examiner's rejection of claim 30 over Mack in view of Turkowski is premised on 
the assertion that one skilled in the art would have been motivated to change the viewpoint 
modification mentioned in col. 5, lines 1-12, to include generation of new viewpoints based 
on the interpolation methods of Turkowski. Contrary to the Examiner's assertion, one skilled 
in the art would not have been motivated to change Mack's teachings in this way. In 
particular, the reference to ''viewpoint modifications" in col. 5, lines 1-12 in Mack merely 
refers to '"commands" that may be input by the user via head or eye movements. In this 
process, the head or eye movements are converted into 3-D input data and interpreted as 
viewpoint modification commands. In accordance with Mack's teachings, the stereo images 
are used only to determine the 3-D input data; they are not presented to the user; indeed, the 
user w ould not have any interest in different views of his or her eye or head movements. 
Therefore, one skilled in the art would not have had any motivation to change the viewpoint 
modification mentioned in col. 5, lines 1-12, to include generation of new viewpoints based 
on the interpolation methods of Turkowski interpolation between the stereo images, as 
proposed by the examiner. 
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For this additional reason, the rejection of claim 30 under 35 U.S.C. §1 03(a) over 
Mack in view of Turkowski should be withdrawn. 

B. Claim 65 

Claim 65 incorporates the elements of independent claim I . Turkowski does not 
make-up for the failure of Mack to disclose or suggest the elements of independent claim I 
discussed above. Therefore, claim 65 is patentable over Mack in view of Turkowski for at 
least the same reasons explained above in connection with independent claim 1 . 

Claim 65 also is patentable over Mack in view of Turkowski for the additional reason 
explained above in connection with claim 30 (namely, one skilled in the art would have been 
motivated to change the viewpoint modification mentioned in col. 5, lines 1-12, to include 
generation of new v iewpoints based on the interpolation methods of Turkowski). 

7. Claims 33-37. 39. and 60-64 

Claims 33-37, 39, and 60-64 stand rejected under 35 U.S.C. § 103(a) over Deleeuw 
(U.S. 6/753,879) in view of Ellenby (U.S. 5,682,332). 

a Independent claim 33 

Independent claim 33 recites: 



Claim 33 (previously presented): A method of 
interfacing with a machine, comprising: 

displaying an image at a display location disposed 
between a viewing space and an interactive space, wherein the 
displayed image is viewable from a perspective in the viewing 
space; 

acquiring images of the interactive space from a field of 
view directed toward the interactive space along an optical axis 
intersecting a central area of the display location; 

detecting an input target in the acquired images; 

computing coordinates of the input target detected in the 
acquired images; 

identifying an input instruction based on the computed 
input coordinates; and 
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executing the identified input instruction on the 



machine. 



The rejection of claim 33 under 35 U.S C. § 1 03(a) over Deleeuw in view of Ellenby 
should be withdrawn because the cited references do not disclose or suggest all the elements 
of the claim. 

For example, neither Deleeuw nor Ellenby discloses or suggests "identifying an input 
instruction based on the computed input coordinates/' as recited in claim 33. 

In the rationale given by the Examiner in support of the rejection of claim 33, the 
Examiner has not shown that Deleeuw in view of Ellenby discloses the "identifying^ element 
of claim 1 (see § 9 on pages 1 1-12 of the final Office action). Instead, the Examiner 
improperly has simply read the 'identifying"' element of claim 33 out of the claim. Thus, the 
Examiner has not established a prima facie case that claim 33 is obvious over Deleeuw in 
view of Ellenby. For at least this reason, the rejection of claim 33 under 35 U.S.C. § 1 03(a) 
over Deleeuw in view of Ellenby should be withdrawn. 

In addition, the Examiner has argued that Deleeuw discloses "executing input 
instructions based on the coordinates of the target in the acquired images (col. 3, lines 29- 
35)" (page 12, lines 4-5, of the final Office action). This assertion, however, does not meet 
the language of claim 33. In particular, claim 33 recites "identifying an input instruction 
based on the computed input coordinates/ 5 Merely "executing input instructions based on the 
coordinates of the target in the acquired image i? does not constitute "identifying an input 
instruction based on the computed input coordinates:" nor does it constitute "executing the 
identified input instruction on the machine." For at least this additional reason, the rejection 
of claim 33 under 35 U.S.C. § 103(a) over Deleeuw in view of Ellenby should be withdrawn. 

Furthermore, neither Deleeuw nor Ellenby discloses or suggests "acquiring images of 
the interactive space from a field of view directed toward the interactive space along an 
optical axis intersecting a central area of the display location," where coordinates of a input 
target detected in the acquired images of the interactive space are computed and based on 
which an input instruction is identified. 

The Examiner has acknowledged that Deleeuw does not disclose the "acquiring"' 
element of independent claim 33 (see page 12, first full % The Examiner has relied on 
Ellenby in an effort to make-up for this failure of Deleeuw's disclosure. In particular, the 
Examiner has taken the position that (page 1 2, second full %): 
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Ellenby discloses an augmented reality display system wherein 
the camera system (Fig. 4, element 41) is location with an 
optical axis (Fig. 4, element 49) intersecting a central area of 
the display location (Fig. 4, element 42). 



With respect to the embodiment relied upon by the Examiner, Ellenby discloses (col. 
6, lines 26-35): 

. . . FIG. 4 shows an arrangement where a camera 41 and a 
display 42 are aligned to a viewing axis 49. Electronic images 
are transmitted 43 to a computer having an electronic image 
combiner 47. Measurements of position 44 and attitude 45 
stimulate an image generator 46 which provides computer 
generated graphics and imagery relating to the scene being 
addressed as predicted by the computer and transmits 48 this 
information to the combiner 47. A composite image is then 
presented at the display aligned with its normal direction along 
the viewing axis. 

Thus, Ellenby's imaging system does not acquire images of "an interaction space' in 
which an input target can be detected, coordinates of the input target detected in the acquired 
images can be computed, and an input instruction can by identified based on the computed 
input coordinates detected in the acquired images of the interactive space. Instead, Ellenby's 
imaging system is designed to capture images of a remote scene (e.g., a scene of objects in 
outer space, or a scene of objects on Earth) that can be uniquely identified by position and 
attitude (see col. 4, lines 55-59). 

For at least this additional reason, the rejection of claim 33 under 35 U.S.C. § 103(a) 
over Deleeuvv in view of Ellenby should be withdrawn. 

The rejection of claim 33 also should be withdrawn because at the time the invention 
was made there was no apparent reason to combine the teachings of Deleeuw and Ellenby in 
the manner proposed by the Examiner. In particular, the Examiner has taken the position that 
(page 12, second full % emphasis added): 



At the time of invention it would have been obvious to one of 
ordinary skill in the art to modify the teachings of Deleeuw 
with the teachings of Ellenby. The reflective glass position of 
Deleeuw used to provide a viewing location of the augmented 
realm- display svstem could be replaced with a full display with 
camera directly behind the display of Ellenby. The motivation 
would be to provide an augmented reality system where the 
final viewed image appears as if the viewer was looking at the 
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scene with augmented information (Ellenby: col. 3, lines 48- 
67). Thus, it would have been obvious to modify the teachings 
of Deleeuw with the teachings of Ellenby to produce a method 
of interfacing with a machine as described in claims 33, 63, and 

64, 

Contrary to the Examiner's position, one skilled in the art would not have been 
motivated to replace the transreflective or dichroic reflector 24 with Ellenby's imaging 
system shown in FIG. 4. 

First, Ellenby's display 42 is opaque and therefore the Examiner's proposed 
modification of Deleeuw's system would defeat the object of Deleeuws invention to enable 
the user to directly view his or her hands (see col. 2. lines 6-1 0; also see claim 1 ). 

Second, the Examiner's proposed modification of Deleeuw ? s system would require a 
display device that was integrated with the camera and therefore, unlike Deleeuvv's invention, 
the Examiner's proposed modification could not leverage a user's existing 
computer/entertainment equipment. In this way, the Examiner's proposed modification 
would defeat the object of Deleeuw's invention to provide ' virtual reality experience 
available at a relatively lower cost, for example in connection with toys and other 
entertainment devices" (col. 1, lines 23-26). 

Third, Ellenby's imaging system is designed to capture images of a remote scene 
(e.g., a scene of objects in outer space, or a scene of objects on Earth) that can be uniquely 
identified by position and attitude (see col. 4, lines 55-59). Accordingly, the Examiners 
proposed modification of Deleeuw" s system would not work for its intended purpose of 
capturing images of the users hand in the enclosure 12 (see FIG. I). 

Fourth, the rationale given by the Examiner in support of his proposed modification of 
Deleeuw's system (i.e., :> to provide an augmented reality system where the final viewed 
image appears as if the viewer was looking at the scene with augmented information") 
already is achieved by Deleeuvv's system (see, e.g., col. 2, lines 6-16). Since Deleeuw 
already achieves the purported result of the Examiner's proposed modification of Deleeuw' s 
system, one skilled in the art would not have had any apparent reason to modify Deleeuw's 
system based on Ellenby's teachings 'Ho provide an augmented reality system where the final 
viewed image appears as if the viewer was looking at the scene with augmented 
information." The fact is that neither Deleeuw nor Ellenby nor the knowledge generally 
available at the time the invention was made would have led one skilled in the art to believe 
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that there was any problem to be solved or any advantage that would be gained by the 
Examiner's proposed modification of Deleeuw 7 s system because such modification would 
not have served any apparent useful purpose. The Examiner's reasoning in this regard only- 
evidences the fact that the Examiner improperly has engaged in impermissible hindsight 
reconstruction of the claimed invention, using applicants' disclosure as a blueprint for piecing 
together elements from the prior art in a manner that attempts to reconstruct the invention 
recited in claim 33 only with the benefit of impermissible hindsight (see KSR Intl Co. v. 
Teleflex Inc.. slip op. at 17: factfinder should be aware, of course, of the distortion caused 
by hindsight bias and must be cautious of arguments reliant upon ex post reasoning."). 
Without any basis for combining the cited references, the Examiner' rationale in support of 
his proposed combination of Deleeuw and Ellenby amounts to no more than an impermissible 
conclusory statement which cannot establish that one skilled in the art would have had any 
apparent reason to combine Fan and Porikli in the manner proposed by the Examiner. See In 
re Kahn, 441 F. 3d 977, 988 (CA Fed. 2006) ("IRjejections on obviousness grounds cannot 
be sustained by mere conclusory statements; instead, there must be some articulated 
reasoning with some rational underpinning to support the legal conclusion of obviousness"). 

For at least these additional reasons, the rejection of independent claim 33 under 35 
U.S.C. § 103(a) over Deleeuw in view of Ellenby should be withdrawn. 

b. Claims 34-37. 39. and 60-62 

Each of claims 34-37, 39, and 60-62 incorporates the elements of independent claim 
33 and therefore is patentable over Deleeuw in view of Ellenby for at least the same reasons 
explained above in connection with independent claim 1. 

8. Claims 41-44 and 46 

Claims 41-44 and 46 stand rejected under 35 U.S.C. § 103(a) over Deleeuw (U.S. 
6,753,879) in view of Ellenby (U.S. 5,682,332) and Mack (U.S. 6,198,485). 

Each of claims 41-44 and 46 incorporates the elements of independent claim 33. 
Mack does not make-up for the failure of Deleeuw in view of Ellenby to disclose or suggest 
the elements of independent claim 33 discussed above. Therefore, claims 41 -44 and 46 are 
patentable over Deleeuw in view of Ellenby and Mack for at least the same reasons explained 
above in connection with independent claim 33. 
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9. Claims 45. 47. and 48 

Claims 45, 47, and 48 stand rejected under 35 U S C. § 103(a) over Deleeuvv (U.S. 
6,753,879) in view of Elienby (U.S. 5,682,332), Mack (U.S. 6,198,485), and Pryor (U.S. 
7,042,440). 

Each of claims 45, 47, and 48 incorporates the elements of independent claim 33. 
Mack and Pryor do not make-up for the failure of Deleeuw in view of Elienby to disclose or 
suggest the elements of independent claim 33 discussed above. Therefore, claims 45, 47, and 
48 are patentable over Deleeuw in view of Elienby, Mack, and Pryor for at least the same 
reasons explained above in connection with independent claim 33. 

10. Claims 47 and 49-59 
a. Introduction 

Claims 47 and 49-59 stand rejected under 35 U.S.C. § 103(a) over Deleeuw (U.S. 
6,753,879) in view of Elienby (U.S. 5,682,332), Mack (U.S. 6,198,485), and Schmalstieg 
(U.S. 6,842,175). 

Each of claims 47 and 49-59 incorporates the elements of independent claim 33. 
Mack and Schmalstieg do not make-up for the failure of Deleeuw in view of Elienby to 
disclose or suggest the elements of independent claim 33 discussed above. Therefore, claims 
47 and 49-59 are patentable over Deleeuw in view of Elienby, Mack, and Schmalstieg for at 
least the same reasons explained above in connection with independent claim 33. 

VIII. Conclusion 

For the reasons explained above, all of the pending claims are now in condition for 
allowance and should be allowed. 

Charge any excess fees or apply any credits to Deposit Account No. 08-2025. 
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Respectfully submitted. 



Date: June 13.2009 /Edouard Garcia. Reg. No. 38.461/ 

Edouard Garcia 
Reg. No. 38,461 

Telephone No.: (650)965-8342 

Please direct all correspondence to: 

Hewlett-Packard Company 

Intellectual Property Administration 

Legal Department, M/S 35 

P.O. Box 272400 

Fort Collins, CO 80528-9599 



Applicant : Nelson Liang An Chang 
Serial No. : 10/764,423 
Filed : Jan 23, 2004 



Attorney's Docket No.: 2003 1 0822- 1 
Appeal Brief dated June 13, 2009 
Reply to final action dated Jan. 26, 2009 



Page : 23 of 34 
CLAIMS APPENDIX 

The claims that are the subject of Appeal are presented below. 

Claim 1 (previously presented): A method of interfacing with a machine, comprising: 
at each of multiple capture times, contemporaneously acquiring a respective image 

from each of multiple fields of view defining an interactive space to create a respective set of 

contemporaneous images; 

detecting an input target in the acquired images; 

computing two-dimensional coordinates of the input target detected in the acquired 

images; 

constructing a spatiotemporal input data structure linking each of the capture times to 
the computed two-dimensional coordinates of the input target in each of the 
contemporaneously acquired images in a respective one of the sets of contemporaneous 
images; 

processing the spatiotemporal input data structure to identify an input instruction; and 
executing the identified input instruction on the machine. 

Claim 2 (original): The method of claim 1 , wherein images of the interactive space 
are acquired from at least one stereoscopic pair of fields of view directed along substantially 
parallel axes intersecting the interactive space. 

Claim 3 (original): The method of claim I, wherein images of the interactive space 
are acquired from at least three different fields of view. 

Claim 4 (original): The method of claim I. wherein detecting the input target in the 
acquired images comprises comparing values of pixels in the acquired images to at least one 
threshold pixel value. 

Claim 5 (original): The method of clai m 4 ; wherein computing coordinates of the 
input target comprises computing coordinates of centroids of respective groups of pixels in 
the acquired images with values greater than the at least one threshold pixel value. 
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Claim 6 (original): The method of claim 4, wherein detecting the input target in the 
acquired images comprises segmenting foreground pixels from background pixels in the 
acquired images. 

Claim 7 (canceled) 

Claim 8 (previously presented): The method of claim 1, further comprising 
computing calibration parameters for the multiple fields of view. 

Claim 9 (original): The method of claim 8, wherein computing coordinates of the 
detected input target comprises computing three-dimensional coordinates of the input target 
in the interactive space based on the computed two-dimensional coordinates and the 
computed calibration parameters. 

Claim 10 (previously presented): The method of claim 9, wherein the spatiotemporal 
input data structure additionally links each of the capture times to respective three- 
dimensional coordinates of the input target computed from the computed two-dimensional 
coordinates of the input target in each of the contemporaneously acquired images in a 
respective one of the sets of contemporaneous images. 

Claim 1 1 (original): The method of claim 1, further comprising acquiring color 
values of the detected input target in the acquired images. 

Claim 12 (previously presented): The method of claim 11, wherein the 
spatiotemporal input data structure additionally links each of the capture times to respective 
color values determined from the contemporaneously acquired images in a respective one of 
the sets of contemporaneous images. 

Claim 13 (original): The method of claim 1, wherein the spatiotemporal input data 
structure is constructed in the form of a linked list of data records. 
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Claim 14 (previously presented): The method of claim 1 , wherein processing the 
spatiotemporal input data structure comprises identifying traces of the input target in the 
interactive space, each trace being defined by a respective set of connected data items in the 
spatiotemporal input data structure. 

Claim 15 (original): The method of claim 14, wherein identifying traces comprises 
detecting state change events and segmenting traces based on detected state change events. 

Claim 16 (original): The method of claim 14, wherein identifying traces comprises 
computing coordinates of bounding regions encompassing respective traces. 

Claim 1 7 (original): The method of claim 16 ? wherein the computed bounding region 
coordinates are two-dimensional coordinates of areas in the acquired images. 

Claim 18 (original): The method of claim 16, wherein the computed bounding region 
coordinates are three-dimensional coordinates of volumes in the interactive space. 

Claim 19 (original): The method of claim 14, wherein the spatiotempora] input data 
structure is processed to interpret the identified input target traces. 

Claim 20 (original): The method of claim 19, further comprising comparing an 
identified trace to a predefined representation of an input gesture corresponding to a 
respective input instruction. 

Claim 21 (original): The method of claim 2() f wherein processing the spatiotemporal 
input data structure comprises translating the trace into a predefined alphanumeric character. 



Claim 22 (original): The method of claim 19, further comprising comparing an 
identified trace to a location in the interactive space corresponding to a virtual interactive 
object. 
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Claim 23 (original): The method of claim 22, wherein the virtual interactive object 
corresponds to a virtual machine instruction input. 

Claim 24 (original): The method of claim 23, wherein the virtual machine instruction 
input is predefined. 

Claim 25 (original): The method of claim 23, further comprising constructing the 
virtual machine instruction input in response to processing of at least one identified input 
target trace. 

Claim 26 (original): The method of claim 23, wherein the virtual machine instruction 
input corresponds to a respective mode of interpreting traces. 

Claim 27 (original): The method of claim 1 ? wherein executing the identified input 
instruction comprises displaying an image in accordance with the identified input instruction. 

Claim 28 (previously presented): The method of claim 27, wherein the displaying 
comprises displaying a combination of image data generated based on the acquired images 
and machine-generated virtual image data 

Claim 29 (original): The method of claim 27, further comprising displaying a 
sequence of images at the display location showing a virtual object being manipulated in 
accordance with one or more identified input instructions. 

Claim 30 (previously presented): A method of interfacing with a machine, 
comprising: 

acquiring sets of contemporaneous images of an interactive space from multiple 
respective fields of view; 

detecting an input target in the acquired images; 

computing coordinates of the input target detected in the acquired images; 
constructing a spatiotemporal input data structure linking input target coordinates 
computed from contemporaneous images to respective reference times; 
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processing the spatiotemporal input data structure to identify an input instruction; and 
executing the identified input instruction on the machine; and 
interpolating between fields of view to generate a synthetic view of the interactive 



Claim 3 1 (previously presented): A system for interfacing with a machine, 
comprising: 

multiple imaging devices configured to contemporaneously acquire, at each of 
multiple capture times, a respective image from each of multiple fields of view defining an 
interactive space to create a respective set of contemporaneous images; and 

a processing module configured to detect an input target in the acquired images, 
compute two-dimensional coordinates of the input target detected in the acquired images, 
construct a spatiotemporal input data structure linking each of the capture times to the 
computed two-dimensional coordinates of the input target in each of the contemporaneously 
acquired images in a respective one of the sets of contemporaneous images, process the 
spatiotemporal input data structure to identify an input instruction, and execute the identified 
input instruction on the machine. 

Claim 32 (previously presented): A machine-readable medium storing machine- 
readable instructions for causing a machine to: 

at each of multiple capture times, contemporaneously acquire a respective image from 
each of multiple fields of view defining an interactive space to create a respective set of 
contemporaneous images; 

detect an input target in the acquired images; 

compute two-dimensional coordinates of the input target detected in the acquired 
images; 

construct a spatiotemporal input data structure linking each of the capture times to the 
computed two-dimensional coordinates of the input target in each of the contemporaneously 
acquired images in a respective one of the sets of contemporaneous images; 

process the spatiotemporal input data structure to identify an input instruction; and 

execute the identified input instruction on the machine. 



space. 



Applicant : Nelson Liang An Chang 
Serial No. : 10/764,423 
Filed : Jan 23, 2004 
Page : 28 of 34 



Attorney's Docket No.: 2003 1 0822- 1 
Appeal Brief dated June 13, 2009 
Reply to final action dated Jan. 26, 2009 



Claim 33 (previously presented): A method of interfacing with a machine, 
comprising: 

displaying an image at a display location disposed between a viewing space and an 
interactive space, wherein the displayed image is viewable from a perspective in the viewing 
space. 

acquiring images of the interactive space from a field of view directed toward the 
interactive space along an optical axis intersecting a central area of the display location; 
detecting an input target in the acquired images: 
computing coordinates of the input target detected in the acquired images; 
identifying an input instruction based on the computed input coordinates; and 
executing the identified input instruction on the machine. 

Claim 34 (original): The method of claim 33, wherein the display location 
corresponds to a display area of a portable electronic device. 

Claim 35 (original): The method of claim 33, wherein the display location 
corresponds to a display area embedded in a desktop surface. 

Claim 36 (original): The method of claim 33, wherein displaying the image 
comprises projecting the image onto a surface. 

Claim 37 (original): The method of claim 33, wherein acquiring images comprises 
acquiring images of the interactive space from at least one field of view disposed betw een the 
display location and the interactive space. 

Claim 38 (canceled) 

Claim 39 (original): The method of claim 33, wherein acquiring images comprises 
acquiring images of the interactive space from multiple fields of view. 
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Claim 40 (original): The method of claim 39, further comprising interpolating 
between fields of view to display an image at the display location corresponding to a 
synthetic view of the interactive space. 



calibration parameters for the multiple fields of view. 

Claim 42 (original): The method of claim 41, wherein computing coordinates of the 
detected input target comprises computing three-dimensional coordinates of the input target 
in the interactive space based on the computed calibration parameters. 

Claim 43 (original): The method of claim 33 ? wherein detecting the input target in the 
acquired images comprises comparing values of pixels in the acquired images to at least one 
threshold pixel value. 

Claim 44 (original): The method of claim 43 ? wherein computing coordinates of the 
input target comprises computing coordinates of centroids of respective groups of pixels in 
the acquired images with values greater than the threshold. 

Claim 45 (original): The method of claim 43 ? wherein detecting the input target in the 
acquired images comprises segmenting foreground pixels from background pixels in the 
acquired images. 

Claim 46 (original): The method of claim 33, wherein computing coordinates of the 
detected input target comprises computing two-dimensional coordinates of the input target 
detected in the acquired images. 

Claim 47 (original): The method of claim 33 ? wherein identifying an input instruction 
comprises identifying traces of the input target in the interactive space. 



Claim 41 (original): The method of claim 39, further comprising computing 



Claim 48 (original): The method of claim 47, wherein identifying traces comprises 
detecting state change events and segmenting traces based on detected state change events. 
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Claim 49 (original): The method of claim 47, wherein identifying traces comprises 
computing coordinates of bounding regions encompassing respective traces. 

Claim 50 (original): The method of claim 49, wherein the computed bounding region 
coordinates are two-dimensional coordinates of areas in the acquired images. 

Claim 51 (original): The method of claim 49, w herein the computed bounding region 
coordinates are three-dimensional coordinates of volumes in the interactive space. 

Claim 52 (original): The method of claim 47, wherein identifying the input 
instruction comprises interpreting the identified input target traces. 

Claim 53 (original): The method of claim 52, further comprising comparing an 
identified trace to a predefined representation of an input gesture corresponding to a 
respective input instruction. 

Claim 54 (original): The method of claim 53, wherein processing the spatiotemporal 
input data structure comprises translating the trace into a predefined alphanumeric character. 

Claim 55 (original): The method of claim 52, further comprising comparing an 
identified trace to a location in the interactive space corresponding to a virtual interactive 
object. 

Claim 56 (original): The method of claim 55, wherein the virtual interactive object 
corresponds to a virtual machine instruction input. 

Claim 57 (original): The method of claim 56, wherein the virtual machine instruction 
input is predefined. 
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Claim 58 (original): The method of claim 56, further comprising constructing the 
virtual machine instruction input in response to processing of at least one identified input 
target trace. 

Claim 59 (original): The method of claim 56, wherein the virtual machine instruction 
input corresponds to a respective mode of interpreting traces. 

Claim 60 (original); The method of claim 33, wherein executing the identified input 
instruction comprises displaying an image at the display location in accordance with the 
identified input instruction. 

Claim 61 (original): The method of claim 60, wherein the displayed image comprises 
a combination of image data generated based on the acquired images and machine-generated 
virtual image data 

Claim 62 (original): The method of claim 60, further comprising displaying a 
sequence of images at the display location showing a virtual object being manipulated in 
accordance with one or more identified input instructions. 

Claim 63 (previously presented): A system of interfacing with a machine, 
comprising: 

a display configured to present an image at a display location disposed between a 
viewing space and an interactive space, wherein the displayed image is viewable from a 
perspective in the viewing space; 

at least one imaging device configured to acquire images of the interactive space from 
a field of view directed toward the interactive space along an optical axis intersecting a 
central area of the display location; and 

a processing system configured to detect an input target in the acquired images, 
compute coordinates of the input target detected in the acquired images, identify an input 
instruction based on the computed input coordinates, and execute the identified input 
instruction on the machine. 
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Claim 64 (previously presented): A machine-readable medium storing machine- 
readable instructions for causing a machine to: 

display an image at a display location disposed between a viewing space and an 
interactive space, wherein the displayed image is viewable from a perspective in the viewing 
space; 

acquire images of the interactive space from a field of view directed toward the 
interactive space along an optical axis intersecting a central area of the display location; 
detect an input target in the acquired images: 

compute coordinates of the input target detected in the acquired images; 
identify an input instruction based on the computed input coordinates; and 
execute the identified input instruction on the machine. 

Claim 65 (original): The method of claim I , further comprising interpolating betw een 
ones of the images contemporaneously acquired from ones of the fields of view to generate a 
synthetic view of the interactive space. 
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EVIDENCE APPENDIX 



There is no evidence submitted pursuant to 37 CFR §§ 1. 130, 1.131, or 1.132 or any 
other evidence entered by the Examiner and relied upon by Appellant in the pending appeal. 
Therefore, no copies are required under 37 CFR § 41.37(c)(l)(ix) in the pending appeal. 
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RELATED PROCEEDINGS APPENDIX 



Appellant is not aware of any decisions rendered by a court or the Board that will 
directly affect or be directly affected by or have a bearing on the Board's decision in the 
pending appeal. Therefore, no copies are required under 37 CFR § 41.37(c)(l)(x) in the 
pending appeal. 



